Assessing Collaborative Problem Solving through Automated Technologies

Today, proficiency in collaborative problem solving (CPS) is necessary for success in both college and workplace, as well as the ability to perform that collaboration in various settings. At the same time, structuring computer-based CPS assessment, specifically for large-scale programs, is challenging. In order to perform standardized assessment of CPS skills on an individual level, each student should be matched with various types of group members, should be tested on a variety of CPS skills, and must apply the skills in varied contexts. One solution to this standardization is to use computer-based agents to serve as the collaborators in the interactions. The aim of this study was to explore students’ performance in human-to-agent (H-A), compared to human-to-human (H-H) CPS assessment tasks. A sample of 14-year olds across three countries participated in the study. Students in both H-H and H-A modes were able to collaborate and communicate by using identical methods and resources. However, while in the H-A mode, students collaborated with a simulated computerdriven partner and in the H-H mode students collaborated with another student to solve a problem. This study is among the first of its kind to investigate systematically students’ CPS performance in H-A and H-H standardized assessment settings. Directions for future research are discussed in terms of their implications to large-scale assessment programs, teaching and learning.